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Operations plays a pivotal role in the success of any human spaceflight program. This 
paper will highlight some of the core tenets of spaceflight operations from a systems 
perspective and use several examples from the Space Shuttle Program to highlight where 
the success and safety of a mission can hinge upon the preparedness and competency of 
the operations team. Further, awareness of the types of operations scenarios and impacts 
that can arise during human crewed space missions can help inform design and mission 
planning decisions long before a vehicle gets into orbit. 

A strong operations team is crucial to the development of future programs; capturing the 
lessons learned from the successes and failures of a past program will allow for safer, 
more efficient, and better designed programs in the future. No matter how well a vehicle 
is designed and constructed, there are always unexpected events or failures that occur 
during space flight missions. Preparation, training, real-time execution, and 
troubleshooting are skills and values of the Mission Operations Directorate (MOD) flight 
controller; these operational standards have proven invaluable to the Space Shuttle 
Program. Understanding and mastery of these same skills will be required of any 
operations team as technology advances and new vehicles are developed. 

This paper will focus on individual Space Shuttle mission case studies where specific 
operational skills, techniques, and preparedness allowed for mission safety and success. 

It will detail the events leading up to the scenario or failure, how the operations team 
identified and dealt with the failure and its downstream impacts. The various options for 
real-time troubleshooting will be discussed along with the operations team final 
recommendation, execution, and outcome. Finally, the lessons learned will be 
summarized along with an explanation of how these lessons were used to improve the 
operational preparedness of future flight control teams. 



NASA/MOD Operations Impacts from Shuttle Program 


Gregory W. Mattes 1 

NASA - Johnson Space Center, Houston, Texas, 77058 

Michael J. FitzPatrick 2 3 
Holly A. Griffith 1 
Michael R. Grabois 4 

United Space Alliance - Johnson Space Center, Houston, Texas, 77058 


This paper will focus on individual Space Shuttle mission case studies where specific 
operational skills, techniques, and preparedness allowed for mission safety and success. It 
will detail the events leading up to the scenario or failure, how the operations team identified 
and dealt with the failure and its downstream impacts. The various options for real-time 
troubleshooting will be discussed along with the operations team final recommendation, 
execution, and outcome. Finally, the lessons learned will be summarized along with an 
explanation of how these lessons were used to improve the operational preparedness of 
future flight control teams. 


I. Introduction 

S ince 1965, members of the National Aeronautics and Space Administration (NASA)’s Mission Operations 
Directorate (MOD) and its predecessor organizations have provided personnel to be flight controllers in Mission 
Control for the Gemini, Apollo, Skylab, Apollo-Soyuz, Space Shuttle, and International Space Station missions at 
the Johnson Space Center (JSC) in Houston, TX. These flight controllers train for hundreds of hours in countless 
simulation sessions to be the “steely-eyed missile men and women” responsible for mission safety and success. This 
training and preparedness has paid off numerous times during the actual missions. 

The philosophy behind training flight controllers is not to have them see every possible malfunction, or even 
every malfunction in which there is a procedure. Rather, they are trained so that they have a good understanding of 
the system and can diagnose an anomaly and identify the failed component) s) - sometimes from the perspective of 
“we have not seen this exact malfunction, but we have seen something similar, so let’s see if that helps us to solve 
this problem” - then determine the impact(s) of the failure, and how (or if) the problem can be solved, worked 
around, or mitigated. This is known on console as the “Failure - Impact - Workaround” (or FIW) philosophy. 

In addition to their technical knowledge of the hardware and system operation, flight controllers are trained to 
master the “soft” skills such as listening and speaking, from being able to monitor several voice communication 
loops to being able to succinctly explain the situation to the Flight Director on console. 

While flight controllers typically have degrees in engineering, it is not a typical engineering design job. There is 
a large focus on teamwork, communication, failure recognition, and failure response. After years of training, flight 
controllers become experts at being able to see when something does not look right in their system; the hard part is 
figuring out what to do when that happens. Flight controllers have to think of all the possible things you can do and 
weigh the pros and cons, which can be intimidating when there are many different options; it often becomes a risk 
vs. risk tradeoff decision that has to be made in a short amount of time. 

Although the spacecraft have changed over the years, the need for those with proper flight controller skills has 
not. This paper focuses on three case studies from the Space Shuttle Program in which the flight controllers were 
able to quickly diagnose and assess the problem, and to use their knowledge and training to present actions for the 
crew to take in order to save the crew and/or vehicle hardware. These events are representative of many other 
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scenarios seen over the 30 years of the Space Shuttle Program, and are used to illustrate the necessity of having 
well-prepared flight controllers during spaceflight operations. 


II. STS-27 Post-Landing Cooling Anomaly 

On December 6, 1988, the Space Shuttle Atlantis completed its third mission, STS-27 (the 27th Shuttle mission 
overall), with a landing at the Kennedy Space Center (KSC) in Florida. Atlantis carried a classified Department of 
Defense (DoD) payload, though the details of this post-landing scenario are not classified. 

While on orbit, the waste heat from the Shuttle’s crew compartment is transferred into water coolant loops, 
which in turn transfer the heat to Freon coolant loops via the Water/Freon Interchanger. The heat is dissipated 
overboard using a combination of radiators mounted inside the payload bay doors and the Flash Evaporator System 
(FES), which sprays water onto the hot Freon Loops. For entry, the radiators are cold-soaked and then bypassed to 
provide a cold sink to be used from approximately 100,000 ft altitude until rollout on the runway. Finally, Ammonia 
(NFI3) Boilers are used during post-landing operations, in which liquid NF13 is sprayed onto the Freon Loops to 
provide cooling until the KSC ground team can connect a Ground Service Equipment (GSE) cooling unit. In order to 
have a safe transition to ground cooling, the NF13 must first be deactivated onboard and confirmed to not be leaking, 
after which the Flight Director coordinates with the ground team to activate ground cooling. Once ground cooling is 
stable, the Mission Control Center (MCC) team officially hands responsibility of the Orbiter over to KSC. 
Additionally, during this ground cooling timeframe, the astronaut crew is egressing the Orbiter and being replaced 
onboard by the Astronaut Support Personnel (ASP), a non-crew astronaut who finishes the post-landing procedures 
for the departing crew. 

Providing proper cooling to the Orbiter is crucial: If cooling is lost, a powerdown of the vehicle must be 
accomplished relatively quickly or the Fuel Cells could overheat and explode, while at the same time temperatures 
can’t be allowed to get too cold for danger of freezing the water at the Water/Freon Interchanger. 

As seen in Fig. 1, the Space Shuttle has two Freon Loops, both of which are typically running at all times. Post- 
landing, the ground Cooling Cart is connected to the Orbiter at the GSE Heat Exchanger, represented by the red 
arrow at top left. The Freon continues clockwise in this schematic, reaching the Ammonia Boiler and the FES before 
splitting into legs cooling the afi coldplates, the payloads, and the water coolant loops. A Flow Proportioning Valve 
can set the flows to be a ratio of either 90% at the Water/Freon Interchanger and 10% at the Payload Heat 
Exchanger, or 57% to 43%. From these legs, the Freon goes through a pump package, heat exchangers for the Fuel 
Cells and midbody coldplates, and finally the radiators before reaching the GSE Heat Exchanger again. The 
Water/Freon Interchanger at far right is where all of the Orbiter’s coolant loops intersect, and if the Freon 
temperature is cold enough it could freeze the water, rupture the interchanger, and cause the complete loss of 
cooling to the Orbiter. 
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A. Detailed Failure Description 

On console in Mission Control for the STS-27 post-landing was EECOM (then known as the Electrical, 
Environmental, and Consumables Management Officer) and in a nearby support room, his Thermal Officer. Much 
like a military hierarchy, MCC’s Flight Director has ultimate authority for the mission and is in charge of the Flight 
Control Room (FCR, or “Front Room”) controllers such as EECOM while the Front Room controllers are in charge 
of the Multi-Purpose Support Room (MPSR, or “Back Room”) controllers (such as TFIERMAL). 

Approximately one hour into the post-landing activities, MCC was in the process of transitioning the cooling 
from onboard resources to the ground Cooling Cart. The second of two NF13 tanks was close to depletion and the 
ground Cooling Cart was ready to be connected to the Orbiter. While EECOM monitored the vehicle cooling, many 
of the other flight controllers had finished monitoring their systems and had already begun their post-landing 
festivities and were preparing to hang the mission plaque in Mission Control to celebrate a successful flight. 

While this was happening around them, TFIERMAL noticed that the NF13 controller had switched over from the 
Primary controller to the Secondary controller. The NH3 controller had undertemp logic that would automatically 
switch from Primary to Secondary in an undertemp condition. Flowever, the NF13 controller is upstream of any 
temperature sensors that are downlinked in the telemetry (the closest were the Evaporator Outlet Temperatures, or 
“Evap Out T”) so it could not immediately be determined as to why the controller switched, and whether it was an 
actual undertemp condition or a controller failure. At typical flow rates in the Freon Loops, it takes about 45 seconds 
for a temperature change at the NF13 controller to reach the Evap Out T sensors, and another 45 seconds for this cold 
mass of Freon to reach the Water/Freon Interchanger. 

After the 45 seconds had elapsed, the Evap Out T sensors plummeted to below 32° F, confirming a true 
undertemp condition and that the Interchanger was in danger of freezing. Both the KSC ground team and the 
EECOM team in the MCC noted this change in the data. 

B. Detailed Description of Real-time Troubleshooting and Resolution 

TFLERMAL’s initial call to EECOM was to deactivate the NH3 system, due to the possibility that the ground 
team had hooked up ground cooling without notifying the team in the MCC, resulting in two cooling systems 
operating at the same time. This configuration could result in an undertemp condition due to a design flaw in the 
NF13 system that allowed a minimum amount of NF13 flow whenever the NH3 controller was activated, even if this 
minimum flow would cause temperatures to be below the control band (35+3°F). The MCC notified the crew 
onboard to deactivate the NF13 controller, and by doing so, EECOM and THERMAL believed they could regain 
temperature control and avoid deactivating the Freon Loops, a more drastic step to prevent the cold Freon from 
reaching the Freon/H20 Interchanger. However, before this call was received onboard the Shuttle, the temperatures 
had already gone colder than their Off-Scale Low (OSL) point of 24.9° F, and the MCC was forced to make the call 
to the crew to turn off both Freon Loops. The Capsule Communicator, or CAPCOM, relayed this call to the crew 
and there was no response, while the 45 -second clock for the cold Freon to reach the Interchanger counted down. 

At this point, THERMAL noted the lack of action by the crew and wondered if there was some discussion 
between EECOM and the Flight Director (both in a different room from THERMAL) over the console and not on 
the communication loops about possible resistance to turning off both Freon Loops. Meanwhile, EECOM assumed 
that the crew would respond quickly to the Freon Loops deactivation call and was reviewing the recovery 
procedures, not realizing that the action to take both Freon Loops off had yet to be performed. EECOM had the 
CAPCOM repeat the call to deactivate the Freon Loops. The Astronaut Support Personnel (ASP, who takes over 
operations inside the crew compartment once the crew departs) responded and turned off the Freon Loops off only 
seconds before the cold slug of Freon reached the Interchanger. 

At this point, the EECOM team in the MCC had stopped the short-term problem of the cold slug reaching and 
rupturing the Water/Freon Interchanger but had introduced a longer term problem. Without Freon Loops running the 
vehicle had no cooling, and without either reestablishing cooling or powering down, the Fuel Cells would overheat 
and explode. Additionally, a long-term power disruption would cause the loss of science experiments and other 
payloads. 

THERMAL and EECOM elected to allow the Interchanger to continue to heat up, as the H20 loops were still 
running and collecting all the cabin equipment heat, and then reactivate a single Freon Loop at a time, knowing that 
they had approximately 10 minutes before the Fuel Cells would start to reach critical temperatures. This decreased 
the efficiency at the Interchanger since only one Freon Loop would initially be running, thereby slowing the transfer 
of heat (or cold in this case) from the Freon Loops to the H20 Loops. When the first Freon Loop was activated, the 
Interchanger was hot enough to absorb the cold slug without incident. Several minutes later, the MCC called the 
crew to reactivate the second loop, and the warm Interchanger was again able to absorb the cold slug without 
incident. 
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In the post-flight debrief of this anomaly, several factors were noted that played into the actions and reactions of 
the flight controllers and onboard personnel: 

• Lack of situational awareness by the non-EECOM members of the Flight Control Team while post- 
landing operations were continuing prior to vehicle handover to KSC. Members of the team had already 
begun to engage in post-landing festivities and the Flight Control Room in the MCC was loud. 

• The mission crew had already exited the Shuttle and the vehicle was manned by the ASP who was not as 
familiar with all the switches in the vehicle as the mission crew. 

• Unbeknownst to the Flight Control Team in the MCC, when the crew exited the vehicle they took all of 
the Flight Data File (crew procedures) with them due to the classified nature of the flight. Thus when 
MCC referred the ASP to a procedure, there were none to be found. 

• At the time, the Flight Rules stated that the MCC in Flouston was in charge of the mission from the point 
where the Shuttle cleared the tower just after liftoff until the crew exited the vehicle. At that time, the 
Ground Team cut off communication from MCC to the Shuttle for a short period and it was likely that the 
first call to deactivate the Freon Loops never made it to the crew or ASP. 

• The KSC Ground Team felt it was urgent to connect the GSE Cooling Cart to the Orbiter and begin 
ground cooling since the Shuttle was on its last cooling system. 

• The KSC Ground Team had also recently changed Cooling Cart procedures so that the ground cooling 
was recirculated inside the Cooling Cart, generating a very cold fluid. When this fluid was introduced into 
the Shuttle GSE heat exchanger it caused the Shuttle Freon Loop temperatures to drop quickly. 

C. How was the Flight Control Team Prepared for the failure? 

Up until this flight, post-landing procedures were not considered a high enough priority item to be trained in 
simulations. It was therefore incumbent upon the EECOM team to mentally walk through the procedures, 
determining the pitfalls and the remediation steps that would be required. 

This was possible because of the extensive systems and flight controller training that both EECOM and 
THERMAL had gone through up to that point. It took over a year to become an Orbit THERMAL and over three 
years to become an Orbit EECOM. Generically, training consists of first mastering the four main tools of flight 
control, 1) Operational drawings, 2) Systems briefs, 3) Flight Rules, and 4) Procedures, and then adding the flight 
controller skills of failure recognition, resolution, next worst failure mitigation, and communication. This training 
starts with workbooks and computer-based training followed by the integrated simulations in the MCC. Mastering 
these skills for a back-room position generally takes over a year. The newly-certified flight controller would then be 
an on-the-job-trainee for several flights to gain real experience before starting in either the Orbit EECOM or in the 
Ascent/Entry back-room flow. Each of these positions requires approximately a year of training, and then training 
begins in the Ascent/Entry EECOM flow. The whole process from new hire to Ascent/Entry EECOM historically 
averaged about 6 years. When this process is complete, the certified flight controller has a wealth of experience both 
in simulations and real-time flight operations to draw from to address the failure scenarios that inevitably arise. 

In the post-flight reviews several items were identified as opportunities for improvement: 

• Better coordination with the KSC ground team: The MCC and KSC established a dedicated 
communication loop and protocol to allow EECOM to talk directly with the cooling team. Previously, all 
communication had to be relayed from EECOM to the Flight Director who would talk to the Convoy 
Commander at KSC who would then relay it to the Cooling Cart team. 

• Better Flight Director awareness of the criticality of the handover of cooling: The Flight Directors were 
briefed on the criticality of the handover of cooling and what the potential dangers were if implemented 
incorrectly. 

• Better situational awareness inside the MCC: To avoid post-landing distraction, activities not directly 
related to landing and vehicle safing (e.g., plaque hanging) were moved off of landing day. 

• Updated Flight Rules: Flight Rule 1-1, which stated that MCC had operational control of the vehicle until 
crew egress, was changed to having control until crew egress or handover to ground cooling, whichever 
occurred last. 

• Additional training: Post-Landing operations and malfunction training were added to simulation sessions 
so that the procedures were well exercised and crew and Flight Control Team familiarized in that phase of 
the mission. 

• Improved procedures: The Evap Out T Low procedure that governed this scenario was reworked so that 
that the Orbiter and Payload hardware would be put in the safest configuration possible. 
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D. Summary 

The Flight Control Team learns something from every mission. In this case, the team learned that even though it 
appears that the flight may be over and it is time to celebrate the team’s accomplishments, failures (man-made or 
otherwise) can still occur. In the midst of distraction the EECOM team identified the undertemp situation, diagnosed 
the cause of two cooling systems running simultaneously, and resolved the situation while preserving crew safety 
and all vehicle hardware. 

III. STS-76 Payload Bay Door Dual Microswitch Failure 

A. General Overview 

STS-76, another flight of the Space Shuttle Atlantis and the 76th Shuttle mission overall, launched in March 
1996. This flight was the third to the Russian Space Station Mir, during which the United States astronauts carried a 
SpaceFlab module in the payload bay. The crew also transferred numerous scientific instruments and payloads to 
Mir in addition to critical items such as water and food. During this mission the U.S. astronauts performed the first 
Extravehicular Activity (EVA) around two mated spacecraft. Atlantis undocked from Mir and the crew completed 
their orbital activities without incident. 

On the intended landing day, the crew began their preparations to reconfigure Atlantis into its entry 
configuration. Approximately 80 minutes into the four-hour-long procedure, the crew closed the Payload Bay Doors 
(PLBDs) per the timeline. Flowever, the weather at the Kennedy Space Center (KSC) was not ideal for landing, so 
the Flight Control Team recommended waving off the landing. After two more orbits the team had decided that it 
was still unsafe to land on that day and that they would try again the next day. The crew was told to perform the 
Deorbit Prep Backout procedure, which would reconfigure Atlantis for orbit operations and open the PLBDs to 
allow for vehicle cooling. (Recall from the previous case that vehicle cooling on orbit is provided by the Freon 
circulating within the radiators on the inside of the doors, supplemented by the Flash Evaporator System (FES) 
which sprays water on the Freon Loops; during the deorbit prep time frame, the radiators are cold-soaked for use on 
entry once the vehicle reaches an altitude where the FES becomes ineffective.) With the PLBDs closed and no 
cooling from the radiators, the Shuttle would eventually run out of spray water for the FES and overheat, requiring 
an emergency deorbit. 

In the Mission Control Center (MCC), the Flight Director has ultimate responsibility for the mission and real- 
time actions of his or her team of flight controllers. The MMACS (Mechanical, Maintenance, Arm and Crew 
Systems) operator works for the Flight Director in the Flight Control Room, with a subordinate called Mechanical 
(or MECH) in a nearby support room. MMACS and MECH are responsible for the real-time operations of the 
Payload Bay Door System, which includes the port and starboard doors, the 16 latches that secure the doors to the 
forward and aft bulkheads, and the 16 centerline latches that secure the doors to each other (with hooks on the port 
door and corresponding rollers on the starboard door). In the normal door opening sequence, the centerline latches 
are opened first (the inner sets of latches 5-8 and 9-12 first, followed by the outer sets of latches 1-4 and 13-16), then 
the starboard forward and aft bulkhead latches, the starboard door, the port forward and aft bulkhead latches, then 
finally the port door. Each mechanism contains two independently powered alternating curent (ac) motors and four 
microswitches, two for each motor representing the range of motion for that mechanism (open and closed, latched 
and released, etc.). The microswitches are wired to the motors such that when the mechanism reaches the end of 
travel, the microswitch is activated (sending telemetry to MCC) and sends a signal to remove ac power from the 
motor. The entire opening and closing sequence is controlled via a series of software commands to the Shuttle’s 
computers on the crew’s display monitors and switches on nearby panels, and can be operated either automatically 
(the crew monitors the Shuttle’s computers operating each item in sequence) or manually by the crew (the crew 
selects an individual mechanism to move). In the Auto Mode, if the mechanism does not reach the intended end of 
travel in the time it would take if only one of the two motors was operating (called “single motor drive time”), the 
computer issues a sequence failure message, annunciates an alert tone to the crew, and stops the automatic sequence. 
In case of a mechanical jam of the mechanism, the motor is designed to slip once the torque reaches a certain value, 
to avoid damaging the motor and/or the mechanism. 
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B. Detailed Failure Description 

During the PLBD opening procedure in the Deorbit Prep Backout on STS-76, both of the Centerline Latch 9-12 
Release microswitches REL A and REL B failed to indicate Open (or Released) after the motors drove for single 
motor time (40 seconds). See Fig. 2 for the location of the latches. This condition resulted in an “S63 PBD SEQ 
FAIL” (Payload Bay Door Auto-Sequence Fail) alarm, which stopped the latch drive. 


Centerline Latch Closeup 




Figure 2. Payload Bay Door Centerline Latch location and hook schematic. 

While the crew was attempting to open the doors on orbit, the MMACS and MECH operators were evaluating 
their telemetry on the ground, which included latch open and closed microswitches, real-time plots of ac currents, 
and the computer alarms. 

The MMACS team quickly concluded that the missing microswitch indications after single motor drive time 
triggered the “S63 PBD SEQ FAIL” alert message. Flowever, the plots of ac current traces indicating the motor 
drive were difficult to interpret. Normally, Payload Bay Door operations drive Centerline Latch Gangs 5-8 and 9-12 
simultaneously, and each Latch Gang has two motors, one driven by ac bus 1 and the other on ac bus 3. Thus, two 
motors drive on each ac bus. It was obvious from the ac plots that both Latch Gangs had initially started driving. The 
open position was indicated at the nominal time for Latch Gang 5-8 with an expected drop in current on both ac 
buses 1 and 3, as the microswitches removed power from the motor at the end of travel. See Fig. 3 for microswitch 
and ac motor traces. 

When the microswitches for latches 9-12 never triggered the open microswitches, ac buses 1 and 3 both 
continued to show a current level that could be expected with either nominal motor drive or the motors driving into 
the built-in slip clutches; the current from Latch Gang 5-8 drive had masked critical current draw information for the 
9-12 gang. With any level of certainty, it was impossible to tell the position of Latch Gang 9-12 using the data in the 
MCC. 
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PLBD Centerline Latch Microswitch Indications 


AC 1 Current 


Close Indications went 
away as Centerline Latch 
Gang 9-12 drove open 


No Release 
Indications for 
Centerline Latch 
Gang 9-12 



Figure 3: Payload Bay Door Centerline Latch Gang 5-8 and 9-12 microswitch and ac motor data 

C. Detailed Description of Impact 

From the data the MMACS team had, the latches were in an indeterminate position. Without all of the PLBD 
centerline Latch Gangs open, it was impossible to open the doors. If it could be determined that the latch was only 
partially open and potentially jammed, it would be possible to slide the latch over the roller when the starboard door 
was opened; however, when the starboard door was re-closed the latch might interfere and prevent door closing and 
a safe vehicle re-entry. 

In the current condition with the doors both closed, there was no radiative cooling, with only the Flash 
Evaporator System (FES) spraying water onto the Freon Loops and a cold sink of Freon trapped in the bypassed 
radiators to provide vehicle cooling. Both, however, are limited consumables: when the cold soaked radiators 
warmed, the vehicle would then be dependent on the finite store of water in the supply water tanks for the FES. In 
this case, the only option would be to re-close all the latches and land somewhere, but the Flight Control Team had 
already passed up the last opportunity for landing at KSC, which was the only landing site called up for that day. 
There was a landing opportunity on the current revolution to Edwards Air Force Base in California, but this 
opportunity required a deorbit burn just thirty minutes after the PLBD latch failure. The next opportunity to a 
primary landing site after that was in another 7-8 orbits, and the water supply for the FES would not last that long. 
While the MMACS team was trying to determine the PLBD situation, the other operators in the room were 
preparing for a potential deorbit and also powering down to reduce the power and cooling load. 

It was clear that the best option was for the MMACS team to determine the position of the centerline Latch Gang 
9-12 so they could determine if it was safe to press forward with PLBD opening and regain vehicle cooling. 

D. Detailed Description of Real-time Troubleshooting and Resolution 

Because the available microswitch and ac current trace data was inconclusive, the MMACS team relied on the 
preplanned procedure to have the crew visually verify the Latch Gang position, and compare it to a drawing in the 
procedure. This requires the crew to look out the aft flight deck windows at the centerline latches toward the aft of 
the vehicle. The MMACS team had the crew working on verifying the latch position while the ground team 
investigated further troubleshooting. The crew had binoculars at their disposal to investigate the position of the 
latch. See Fig. 4 for view out aft flight deck window. 
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Figure 4: View from Port Aft Flight Deck Window with Payload Bay Doors closed. 


This mission carried a single module SpaceHab in the payload bay. The SpaceHab’s main purpose was to 
provide increased habitable volume for storage and science payloads, connected to the crew cabin via the external 
airlock. On the top of the SpaceHab was a viewport window that fortuitously was directly underneath the 9-12 Latch 
Gang. See Fig. 5 for a photo of the SpaceHab within the Shuttle’s payload bay; each gang of latches (1-4, 5-8, 9-12, 
and 13-16) occupies one of the four radiator panels that line the payload bay doors from forward to aft. 



Figure 5: View of Shuttle showing location of SpaceHab viewport. 

Latch Gang in the open direction to confirm they were no longer moving open and 


From the aft windows of 
the flight deck, the crew 
reported that Latch Gang 9-12 
appeared to be open, but due to 
the distance away from the 
window, they were not entirely 
certain, even using binoculars. 
The MMACS team asked the 
crew to ingress the SpaceHab 
module (which required 
opening the hatch from the 
middeck into the airlock) and 
look out the overhead 
viewport. The crew reported 
that the 9-12 Latch Gang was 
definitely open; they 
confirmed this by visually 
comparing to Latch Gang 5-8 
which was already confirmed 
to be in the open position and 
to a diagram in an orbiter 
systems data book. The 
MMACS team then had the 
crew drive the 9-12 centerline 
as a last attempt to trigger the 
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open microswitches. As expected, the crew reported no visual movement of the latches, and the microswitches did 
not engage. 

Finally, the MMACS team was confident with the position of the Latch Gang and were comfortable pressing 
forward with the remainder of the door opening procedure. The crew then opened the remainder of the centerline 
latches and the starboard bulkhead latches to verify that the starboard door popped open slightly as it does per 
design. They then opened the doors themselves and re-initiated radiative cooling to space. 

Several hours later, the two failed centerline latch 9-12 release microswitch indications recovered and showed 
the nominal open position. This indicated the loss of these indications was likely due to thermal effects and/or 
misrigging, which had been seen on previous flights. 

E. How was the Flight Control Team Prepared for the failure? 

The MMACS team was highly prepared for such a failure. The MMACS flight controller has years of training 
and system support as both a MECH controller and a MMACS trainee before being certified to work in the primary 
Flight Control Room. They prepare through studying electrical and mechanical schematics, reading and updating 
systems briefs, helping to troubleshoot real-world ground hardware problems, spending countless hours in flight-like 
simulations, and creating operations products prior to every Shuttle flight. In this particular case, while the dual- 
microswitch failure had not been seen in flight or on the ground before (most likely due to the thermal effects of the 
unique radiator coldsoak), the MMACS team had documented single microswitch failures dozens of times over the 
history of the Shuttle program. Additionally, the team knew where the viewport was located on the SpaceHab, 
which allowed the crew to verify the position of the latches. The viewport location had been documented but had not 
been incorporated into any procedure for troubleshooting, and MMACS was able to use his systems knowledge 
outside of the procedures to come up with a time-critical plan. The preplanned procedures drove the crew to visually 
verify the position of the latches. The drawings on board the Shuttle and in the Flight Control Room allowed for the 
proper diagnosis of the latch position. 

This failure required the coordination and cooperation of multiple members of the Flight Control Team working 
seamlessly together to solve a complicated problem was a result of the hard work and dedication of the Shuttle flight 
controllers. 

F. Post Flight Work and Lessons Learned 

As a result of this failure, the Deorbit Prep Backout procedures were changed to separate the opening of 
centerline Latch Gangs 5-8 and 9-12 by driving them sequentially instead of simultaneously. Since both Latch 
Gangs drive one motor each on ac buses 1 and 3, by opening the Latch Gangs separately the currents are not 
overlaid and can be monitored individually to verify if the Latch Gang actually drove to the release position. 

Additionally, the PLBD Opening steps in the Backout procedures were moved earlier in the timeline to give 
more troubleshooting time should a failure occur. This would allow for more time to determine PLBD health and 
status as well as to target additional landing sites should one be needed. 

G. Summary 

The MMACS team used their operational expertise that day to safely open the Payload Bay Doors and avoid a 
potential emergency deorbit by the Space Shuttle, using all the resources on the ground as well as all the crew 
resources on-board. In the meantime, the remainder of the Flight Control Team was preparing for all potential 
outcomes by targeting new landing sites and powering down electrical equipment that required radiator cooling. It 
was a prime example of the entire team acting in unison to solve a complicated and time critical problem. 

III. STS-129 Cryogenic Oxygen Destratification Event 


A. General Overview 

The Space Shuttle Atlantis was in orbit, docked to the International Space Station (ISS) in November 2009 on 
mission STS-129, the 129th flight of the Space Shuttle program, also designated as ISS ULF3 (the third Utilization 
and Logistics Flight). Among those on console in Mission Control Center was the EGIL flight controller, with EPS 
in a support role in a nearby room. 

The EGIL (Electrical Generation and Illumination) and EPS (Electrical Power Systems) flight controllers 
manage and oversee the Space Shuttle’s entire Electrical Power System, from generation to distribution. The Shuttle 
uses three Fuel Cells to generate all of the needed electricity by conversion of cryogenic oxygen (02) and hydrogen 
(H2) stored in tanks in the payload bay, with the resulting electricity distributed throughout the Orbiter via a series 
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of electrical buses. The water that is the byproduct of the Fuel Cell’s electrical generation is contained in a set of 
tanks, to be used for crew consumption, transfer to the 1SS, or as a last resort dumped overboard if not needed. 

During Shuttle flights, the EGIL team spends most of their time managing the limited amount of energy onboard 
the Shuttle to complete the mission. At launch, the EGIL team has a plan of how to use their limited energy. When 
the mission needs re -planning, or days are added to the timeline, EGIL must determine the flight’s energy 
capabilities. 

Once EGIL operators are trained on how to manage vehicle energy, the rest (and the majority) of their training is 
aimed at developing the ability to identify and respond to off-nominal events and failures. The response requires a 
flight controller to be able to think of all possible actions that can be performed to better the situation following the 
event, to think about the risks of doing each action, and to understand how the situation may change with time. 

Given the opportunity, the flight controller can perform a benefit versus risk trade study on each action to determine 
the best course of action given the time constraints. Skills like this, along with communication, prioritization, 
vehicle systems knowledge, and the ability to stay calm under pressure and to plan ahead, are developed through 
training. These skills are important between the EGIL/EPS team, the other flight controllers in MCC, the Flight 
Director, and the supporting engineers. 

B. Detailed Failure Description 

A supply water dump had been planned prelaunch to occur sometime during the middle of the flight, but the 
dump and the accompanying attitude maneuver were later cancelled due to real-time requirements and crew usage. 
This meant that without any significant maneuvers of the Orbiter, there would be no shakeup of the cryogenic 02 
and H2 tanks for six days. Typically, the jet firings of the Orbiter during attitude maneuvers are sufficient to mix the 
cryo inside the tanks. 

In producing the electricity, the Fuel Cells demand cryo and draw from the tanks, and heaters inside the tanks 
keep the pressure within a certain range to accommodate the demand. As the heaters cycle on and off to get cryo out 
of tanks, layers of different densities form in the tank due to the presence of microgravity. This is called 
stratification and is normally not a problem until a maneuver or other shakeup activity occurs to enable cryogenic 
mixing. This “shaking up” is destratification, or “destrat”. 

Regions of cryo with different densities mixing together in microgravity leads to different temperatures mixing, 
and thus the pressure in the tank changes and typically drops. Flight Rules govern the use of the cryo heaters at 
certain pressures: if the pressure drops below a certain value called critical pressure (731 psi for 02), it becomes 
risky to operate the heaters as the cryo is likely to turn from a liquid phase into a gas phase. The EGIL team tries to 
avoid using the cryo heaters below this critical pressure, as localized overheating of the heater elements in a gaseous 
environment could ignite the Teflon in the 02 tank, leading to an Apollo 13 type explosion. With the heaters in 
Auto, the normal operating pressure for the 02 tanks on board the Shuttle is 81 1 to 846 psi for Tanks 1 and 2, and 
840 to 870 psi for Tanks 3, 4, and 5. In NASA vehicles prior to Apollo 13 that used Fuel Cells, the cryo tanks 
contained motors to stir the cryo within to prevent destratification. These stir motors were removed after the 02 tank 
exploded on Apollo 13 in 1970, and were never implemented on the Space Shuttle’s tanks, and thus there is no 
longer any way to prevent destratification in Shuttle cryo tanks. However, it is possible to mitigate the effects by 
knowing when to expect destratification, and to attempt to keep the pressure above the critical pressure. If the 
pressure drops too low, the EGIL team must wait until ambient heating raises the pressure, a process which may take 
several days and leaves the tank unusable (pressure too low) until then. The value of “too low” is dependent upon 
numerous real-time parameters, but one option is to turn the heaters on even though the pressure is below the critical 
pressure. 

Though the stirrers are no longer in the tanks, shakeups still occur from other routine attitude maneuvers and 
burns during the mission. The longer the duration between maneuvers or burns on the Shuttle which mix the cryo 
within the tanks, the more drastic the destratification events tend to be. Operations while docked to the ISS tend to 
be very quiescent, and the tanks do not see much mixing during docked operations. 

C. Detailed Description of Impact 

On STS-129, the team was expecting a rather large destratification event since it had been several days since the 
previous mixing. They were not, however, expecting the widespread destrat that was seen, across all 5 of the 02 
tanks due to the maneuver to the water dump attitude. 

Typically a pressure drop will be seen across one or two tanks; to see all five was alarming and had never been 
seen before. Predicting which tanks would be expected to drop and how much they recover is difficult. The team 
expected Tanks 3 and 4 to drop due to their quantity being closest to a value known to cause trouble during destrat 
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events, and also Tanks 1 or 2 depending on which tank had been used overnight, as that tank would have more 
destrat since it had more heat put into it from the heaters most recently. 

D. Detailed Description of Real-time Troubleshooting and Resolution 

To prepare for the expected destrat event, the EG1L team called for the crew to take the heaters in 02 Tanks 4 
and 5 from OFF to Auto. In this configuration, the heaters would come on automatically once the pressure dropped 
below the lower operating limit of 840 psi. However, they saw pressures drop in all five tanks as all tanks began to 
destratify. 

Heaters in Tanks 3 (already in Auto) and 4 activated when the pressure fell below 840 psi, but could not keep up 
with the rapidly falling pressure. Only the pressure in Tank 5 stabilized with heaters. The EGIL team began to 
consider whether the heaters in 3 and 4 should remain on below the approaching critical pressure, and ultimately 
decided to take tanks 3 and 4 to OFF. 

See Fig. 6 for a plot of cryo pressure vs. time. 



Figure 6: Cryo 02 Tank Pressure vs. Time 

Managing cryo destrat events is closer to an art form than a science for the EGIL team due to the limited insight 
into the conditions inside the tanks (two pressure sensors located near each other, one fluid temperature sensor, and 
one heater temperature sensor for each of the two tank heaters). Furthermore, the tank has a significant thermal 
inertia, so there is a lag between heater activation and tank pressure response, thus the team has to be able to 
reasonably predict the tank’s response in advance. 

From past flight experience, the EGIL team thought that there was a good chance that the pressure was going to 
continue to decrease, so to safe the situation they called for the crew to deactivate the Tank 3 and 4 heaters. By 
watching the pressure decay rates and knowing the tank quantities, the EGIL team knew which tanks to take off and 
could tell which tanks were more heavily stratified. 

After safing the system by being able to provide at least one cryo tank in the immediate term, the EGIL team 
next had to plan for future timelined events, such as crew sleep that night. During the crew sleep period, the Fuel 
Cell and cryo plumbing is set up to protect a potential leak by isolating one cryo manifold from the other two, thus 
requiring the use of two tanks. The team had to balance not wanting to operate heaters below the critical pressure, 
how to manage the sleep period, and how to manage the cryo heater rotation and tank depletion for the rest of the 
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mission. See Fig. 7 for a schematic of the cryo/Fuel Cell 
manifold, showing the three Fuel Cells (FC), the locations 
of the manifold valves, supply valves (which provide 
oxygen to the cabin for crew breathing via the 
Environmental Control System), and the reactant valves 
(which provide 02 and H2 to the Fuel Cells). 

At approximately 56% quantity in tanks 1-4, the two 
phase saturation pressure (or dome) was 660 psi, well 
below the critical pressure of 73 1 psi but more 
importantly below the current tank pressure. See Fig. 8 
for the isotherm plot of cryo 02 temperature, pressure, 
and quantity from the EGIL Console Handbook used in 
MCC to determine critical pressure. 

Oaygen tank isotherms. *F 



Figure 8: Isotherm plot of cryo 02 temperatures 
relative to pressure and quantity. 


Figure 7: Cryo and Fuel Cell manifold schematic 

The EGIL team determined that the limited 
amount of usable fluid in tank 5 (the only tank T * n< 

whose pressure was above the critical pressure) Figure 8: Isotherm plot of cryo 02 temperatures 

would not last the night, and that they needed to relative to pressure and quantity, 

activate heaters in another tank to have enough 
usable fluid to supply the Fuel Cells with reactants 

needed to make electricity, and found themselves in a risk/trade case. They knew they could either take heaters on 
immediately and raise pressure, thereby exposing themselves to the risk of having heaters on in a tank that is below 
the critical pressure, or they could wait and hope that natural heat leak from outside the tank would cause the 
pressure inside the tank to slowly rise, but fast enough to be able to be usable before Tank 5 ran dry. However, the 
team could not know when that would start happening, and knew they would have to research past flight data for 
destrat events to be able to predict how the tank pressures would react for various heater configurations and changes 
in orbiter attitude. The team had to make their best decision given the information available at the time. 

Shuttle Flight Rules allowed the cryo heaters to be turned on even if the pressure in a tank had dropped below 
73 1 psi, as long as the pressure is not below the ‘‘dome”, a term related to the curve of a plot of pressure related to 
quantity and the liquid/gas phase. At a particular quantity and pressure, it can be determined how much would be 
liquid or gas, with the tanks at the highest quantity having the lowest critical pressure. In this case the team 
concluded that while Tank 2 had gone below 731 psi, it had the highest pressure of the four tanks and it had not 
gone below its dome pressure of 660 psi. With Tank 2 still in the two-phase region, the team felt comfortable taking 
these heaters back on, and it would also allow a cryo manifold to stay closed for crew sleep. 

The decision that the EGIL team made to use Tank 2 was something that needed to be done quickly because the 
pressure was continuously going down in that tank. Had they done nothing, it was likely that the pressure would 
have gone below the dome, after which they would not have been allowed to use the heaters in that tank and would 
have had to wait on natural heat leak to raise it again. 

Had the pressures dropped below the dome on the other tanks, the team would have had to stay on only Tank 5 
and hope that heat leak would raise the pressures back above the dome before Tank 5 depleted. If tank 5 had been 
depleted, the crew would have had to activate the tank heaters in an even riskier situation. 
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E. How was the Flight Control Team Prepared for the failure? 

Listening to the recorded voice loops after the flight, it is evident by the calmness in their voices that the EG1L 
team’s training played a large role in how they handled this situation: at no point during the recording did they seem 
nervous or unsure of themselves. During simulations, the team sees many failures, frequently worse than this one, 
and the philosophy behind flight controller malfunction training is to desensitize them to stress by increasing the 
difficulty level of failures as training progresses towards certification. If the flight controller can stay calm in a high- 
stress situation, then they can remain calm during situations on a real flight. Flight Controllers are trained to deal 
with the Failure - Impact - Workaround way of thinking: deal with the immediate failure; what are the immediate, 
short-term, and long-term impacts; and begin to plan for the next worst failures that could occur. 

In this scenario, the EGIL team wanted to make sure that the crew would not have to be woken up during their 
overnight sleep to manage the cryo system. They efficiently used their console tools to determine what needed to be 
done before crew sleep, and in which tanks it was safe to activate the heaters. This allowed them a bit of luxury to 
critique themselves in real-time as the situation was unfolding, in terms of which cryo tank(s) to use. A delay in 
relaying the plan to the Flight Director would have meant possibly losing the opportunity to reconfigure heaters if 
needed prior to crew sleep. 

F. Post Flight Work and Lessons Learned 

Even with the potentially dangerous situation that the EGIL team faced on the ground, there was no change in 
philosophy on how to manage a similar case in the future, and in the post-flight debrief the team and their managers 
generally supported the Flight Control Team’s actions on console at the time. Some flight controllers recommended 
leaving the heaters off and let the heat leaking back into the tanks raise the pressure before the crew went to sleep. In 
either case, however, the EGIL team would have to take the heaters on below the critical pressure as the heat 
leakage would not support raising the pressure above the critical pressure before sleep. 

G. Summary 

In conclusion, cryo tank destratification is something that is seen regularly on Space Shuttle flights. On STS-129 
this destratification was much worse due to being docked to the Space Station for a longer duration with no 
maneuvers performed to shake up the tanks. Although flight controller training does include specific unexpected 
failures, flight controllers are also prepared to react and think in certain ways that allow them to work through 
failures. This particular event was a combination of these two types of failures: destratification, which the team was 
prepared for and had seen before, combined with the unexpected severity of the destratification. The methodology 
that the flight controllers were trained to use to think through all kinds of failures helped them to react quickly while 
still keeping in mind the flight rules, timeline, and other technical aspects needed to operate safely. 

In hindsight, the two controllers who were on console for this event acknowledge that there were things they 
could have done differently but each action had its pros and cons. Ultimately they admitted that they would not 
have changed their decisions and were able to give rationale as to why. As the event was unfolding they also talked 
through each potential action thoroughly as possible, as they had been trained to do. 

IV. Conclusion 

It is important to be able to realize that in human spaceflight operations, most off-nominal events will not be 
black and white. There will be a gray area where a multitude of possibilities could be the right answer. It is 
necessary to understand that flight control is not about memorizing a list of procedures for given failures. While 
those failures exist and this is a part of the job, these are not the majority of situations that flight controllers train for. 
The important part of flight control is to be able to handle the unexpected situation and the thought process that 
leads a controller to the right decision and to be back up that decision with sound engineering judgment and 
technical data. This skill can only be learned through good training, situational awareness, communication, and 
systems knowledge. 
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STS-27, POST-LANDING COOLING 
ANOMALY 



Shuttle Thermal Basics 

While on orbit, the waste heat from the Shuttle's crew 
compartment is transferred into water coolant loops, which in turn 
transfer the heat to Freon coolant loops via the Water/Freon 
Interchanger. The heat is dissipated overboard using a combination 
of radiators mounted inside the payload bay doors and the Flash 
Evaporator System (FES), which sprays water onto the hot Freon 
loops. 

For entry, the radiators are cold-soaked and then bypassed to 
provide a cold sink to be used from approximately 100,000 ft 
altitude until rollout on the runway. 

NH3 boilers are used during post-landing operations, in which liquid 
NH3 is sprayed onto the Freon loops to provide cooling until the 
KSC ground team can connect a Ground Service Equipment (GSE) 
cooling. 

In order to have a safe transition to ground cooling, the NH3 must 
first be deactivated onboard and confirmed to not be leaking, after 
which the Flight Director coordinates with the ground team to 
activate ground cooling. 

Once ground cooling is stable, the Mission Control Center (MCC) 
team officially hands responsibility of the Orbiter over to KSC. 



Shuttle Thermal Basics con't 

Providing proper cooling to the Orbiter is crucial: 

- If cooling is lost, a powerdown of the vehicle must be 
accomplished relatively quickly or the Fuel Cells could overheat 
and explode, 

- while at the same time temperatures can't be allowed to get 
too cold for danger of freezing the water at the Water/Freon 
Interchanger. 

As seen in the figure below, the Space Shuttle has two Freon loops, 
both of which are typically running at all times. 

Post-landing, the ground cooling cart is connected to the Orbiter at 
the GSE Heat Exchanger, represented by the red arrow at top left. 

The Freon continues clockwise in this schematic, reaching the 
Ammonia Boiler and the FES before splitting into legs cooling the 
aft coldplates, the payloads, and the water coolant loops. 

The Water/Freon Interchanger at far right is where all of the 
Orbiter's coolant loops intersect, and if the Freon temperature is 
cold enough it could freeze the water, rupture the interchanger, and 
cause the complete loss of cooling to the Orbiter. 



Shuttle Thermal Control System 


HYDRAULIC HEAT 
EXCHANGER 


PROPORTIONING 
MODULES (25^) 





A Problem Observed 

Approximately one hour into the post-landing activities, MCC was in 
the process of transitioning the cooling from onboard resources to 
the ground cooling cart. The second of two NH3 tanks was close to 
depletion and the ground cooling cart was ready to be connected to 
the Orbiter. 

While EECOM monitored the vehicle cooling, many of the other 
flight controllers had finished monitoring their systems and had 
already begun their post-landing festivities and were preparing to 
hang the mission plaque in Mission Control to celebrate a 
successful flight. 

While this was happening around them, THERMAL noticed that the 
NH3 controller had switched over from the Primary controller to 
the Secondary controller. The NH3 controller had undertemp logic 
that would automatically switch from Primary to Secondary in an 
undertemp condition. 

At typical flow rates in the Freon loops, it takes about 45 seconds 
for a temperature change at the NH3 controller to reach the Evap 
Out T sensors, and another 45 seconds for this cold mass of Freon 
to reach the Water/Freon Interchanger. 



Undertemp! 

After the 45 seconds had elapsed, the Evap Out T sensors plummeted to 
below 32° F, confirming a true undertemp condition and that the 
Interchanger was in danger of freezing. Both the KSC ground team and the 
EECOM team in the MCC noted this change in the data. 

THERMAL's initial call to EECOM was to deactivate the NH3 system, due to 
the possibility that the ground team had hooked up ground cooling 
without notifying the team in the MCC resulting in two cooling systems 
operating at the same time. 

This config could result in an undertemp condition due to a design flaw in 
the NH3 system that allowed a minimum amount of NH3 flow whenever 
the NH3 controller was activated, even if this minimum flow would cause 
temperatures to be below the control band (35±3 g F). The MCC notified 
the crew onboard to deactivate the NH3 controller, and by doing so, 
EECOM and THERMAL believed they could regain temperature control and 
avoid deactivating the Freon loops, a more drastic step to prevent the cold 
Freon from reaching the Freon/H20 Interchanger. 

However, before this call was received onboard the Shuttle, the 
temperatures had already gone colder than their Off-Scale Low (OSL) point 
of 24.9° F, and the MCC was forced to make the call to the crew to turn off 
both Freon loops. The CAPCOM relayed this call to the crew and there was 
no response, while the 45-second clock for the cold Freon to reach the 
Interchanger counted down. 



Troubleshooting 

At this point, THERMAL noted the lack of action by the crew and 
wondered if there was some discussion between EECOM and the 
Flight Director (both in a different room from THERMAL) over the 
console and not on the communication loops about possible 
resistance to turning off both Freon loops. 

Meanwhile, EECOM assumed that the crew would respond quickly 
to the Freon loops deactivation call and was reviewing the recovery 
procedures, not realizing that the action to take both Freon Loops 
off had yet to be performed. EECOM had the CAPCOM repeat the 
call to deactivate the Freon loops; the ASP responded and turned 
off the Freon loops off only seconds before the cold slug of Freon 
reached the Interchanger. 

So, the EECOM team in the MCC had stopped the short-term 
problem of the cold slug reaching and rupturing the Water/Freon 
Interchanger but had introduced a longer term problem. Without 
Freon loops running the vehicle had no cooling; and without either 
reestablishing cooling or powering down, the Fuel Cells would 
overheat and explode. Additionally, a long-term power disruption 
would cause the loss of science experiments and other payloads. 



Crisis Averted! 

THERMAL and EECOM elected to allow the Interchanger to continue 
to heat up, as the H20 loops were still running and collecting all the 
cabin equipment heat, and then reactivate a single Freon loop at a 
time, knowing that they had approximately 10 minutes before the 
fuel cells would start to reach critical temperatures. 

This decreased the efficiency at the Interchanger since only one 
Freon Loop would initially be running, thereby slowing the transfer 
of heat (or cold in this case) from the Freon Loops to the H20 
Loops. When the first Freon loop was activated, the Interchanger 
was hot enough to absorb the cold slug without incident. 

Several minutes later, The MCC called the crew to reactivate the 
second loop, and the warm Interchanger was again able to absorb 
the cold slug without incident. 

In the post-flight debrief of this anomaly, several factors were noted 
that played into the actions and reactions of the flight controllers 
and onboard personnel. 



Factors noted and fixed since the Incident 


Lack of situational awareness by the non-EECOM members of the flight control 
team while post-landing operations were continuing prior to vehicle handover 
to KSC. Members of the team had already begun to engage in post-landing 
festivities and the flight control room in the MCC was loud. 

The mission crew had already exited the Shuttle and the vehicle was manned 
by the Astronaut Support Personnel (ASP) who was not as familiar with all the 
switches in the vehicle as the mission crew. 

Unbeknownst to the Flight Control Team in the MCC, when the crew exited 
the vehicle they took all of the Flight Data File (crew procedures) with them 
due to the classified nature of the flight. Thus when MCC referred the ASP to a 
procedure, there were none to be found. 

At the time, the Flight Rules stated that the MCC in Houston was in charge of 
the mission from the point where the Shuttle cleared the tower just after 
liftoff until the crew exited the vehicle. The Ground Control team cut off 
communication from MCC to the Shuttle for a short period and it was likely 
that the first call to deactivate the Freon loops never made it to the crew/ASP. 

The KSC ground team felt it was urgent to connect the GSE cooling cart to the 
Orbiter and begin ground cooling since the Shuttle was on its last cooling 
system. 

The KSC ground team had also recently changed cooling cart procedures so 
that the ground cooling was recirculated inside the cooling cart, generating a 
very cold fluid. When this fluid was introduced into the shuttle GSE heat 
exchanger it caused the shuttle Freon Loop temperatures to drop quickly. 



Backup 



Evap Out Temperatures 




STS- 7 6, PAYLOAD BAY DOOR DUAL 
MICROSWITCH FAILURE 



Overview 


Crew closed PLBDs to prepare for Entry 

— Weather was poor, so team waived off for the day 

During PLBD opened (performed to regain vehicle 
cooling for orbit ops) centerline latch gang 9-12 
appeared to never open 

— Two microswitches indicated latch did not reach end 
of travel. 

Control team was in a time critical situation 

— Could the PLBDs be opened to regain cooling? 

— Did the Shuttle require an emergency deorbit? 




Centerline latches 

5_8 9-12 13-16 


Centerline Latch Closeup 




Failure Description 


PLBD centerline latch gang 9-12 

— Came off two closed microswitches 

— Did not receive two open microswitches 

— Alternating current traces were inconclusive 

• Latch gang 5-8 and 9-12 were driven simultaneously 
- Latch gang 5-8 reached end of travel at the expected time 

PLBDs cannot be opened with one centerline 
latch gang closed 

Potentially unsafe to open PLBDs with centerline 
latch gang at undetermined position 

NEED TO DETERMINE THE LATCH POSITION 



PLBD Centerline Latch Microswitch Indications 


Close Indications went 
away as Centerline Latch 
Gang 9-12 drove open 


No Release 
Indications for 
Centerline Latch 
Gang 9-12 . 


AC 1 Current 




ACCurrents 
dropped to half 
magnitude as 
Centerline Latch 
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Failure Impact 


Unable to maintain vehicle cooling for 
extended periods with PLBDs closed 

— Flight control team was performing vehicle power 
downs to reduce heat output 

Potential emergency deorbit 

— Edwards Airforce base 

— Secondary landing site 



Real-time Troubleshooting 


Team was unable to determine latch position 
with microswitch data 

— Latch may have been fully open with two microswitch 
failures 

— Latch may have been at a position mid travel 

AC trace data was inconclusive 

— Current magnitude dropped to approximately 50% 
magnitude when latch gang 5-8 reached end of travel 

• Latch gang slip clutch may have been slipping for entire drive 
time 

• Latch may have starting slipping half way through drive 

• Latch may have been driving the entire time 



Resolution 


Team needed another way to determine the latch 
position 

— Looked at back windows 

— Ingressed the SPACEHAB module to look out the viewport 

• The 9-12 latch gang was directly above viewport 

— Used preplanned visual verification procedures and aids on 
board to verify position 

Using multiple visual cues, the crew was able to 
determine the latch gang was in the fully open position 

— Problem was a dual microswitch failure 

— Crew was able to open the PLBDs and regain radiative 
vehicle cooling 






Lessons Learned 


Microswitch failures were most likely due to the 
radiator cold soak during deorbit preparation 
procedures 

— Microswitch failures were more common during deorbit 
prep PLBD opened versus PLBD opening in the post 
insertion timeframe 

PLBD opening procedures were changed to separate 
the drive of the first two latch gangs 

- Allowed for the AC traces to be interpreted 

PLBD opening was moved to beginning of the deorbit 
prep backout procedure to allow for more 
troubleshooting time should another failure occur 



STS-129, CRYOGENIC OXYGEN 
DESTRATIFICATION EVENT 
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Figure 3. 1-4. Cutaway of cryogenic 0 2 tank assembly 
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